Weak convergence

Convergence in distribution

Definition 1 (convergence in distribution). A random sequence X:\Omega\to\R^\N defined on a probability space (\Omega, \sF, P) converges in distribution to a random variable X_\infty:\Omega'\to\R defined on a probability space (\Omega', \sF', P') if \lim_nF_{X_n}(x) = F_{X_\infty}(x) at all continuity points x of F_{X_\infty}. Convergence in distribution is denoted by \lim_nX_n = X_\infty in distribution.

Proposition 2. Consider a random sequenceX:\Omega\to\R^\N defined on a probability space (\Omega, \sF, P) and a random variable X_\infty: \Omega'\to\R defined on another probability space (\Omega', \sF', P'). Then the following statements are equivalent.

1. \lim_nX_n = X_\infty in distribution.

2. \lim_n\E[g(X_n)] = \E[g(X_\infty)] for any bounded continuous Borel measurable function g:\R\to\R.

3. Characteristic functions converge point-wise, i.e. \lim_n\Phi_{X_n}(u) = \Phi_{X_\infty}(u) for each u \in \R.

Proof. Proof. Let X:\Omega\to\R^\N be a sequence of random variables and let X_\infty:\Omega'\to\R be a random variable. We will show that (a) \implies (b) \implies (c) \implies (a).

1. Applying the bounded convergence theorem to any bounded continuous Borel measurable function g:\R\to\R, we have \lim_n\int_{x \in \R} g(x)dF_{X_n}(x) = \int_{x \in \R} g(x)\lim_ndF_{X_n}(x).

2. Taking g(x) = e^{jux}, we get the result.

3. The proof of this part is technical and is omitted.

 ◻

Example 3 (Convergence in distribution but not in probability). Consider a sequence of non-degenerate continuous random variables X:\Omega\to\R^\N and independent random variable Y: \Omega'\to\R, all with the common distribution F_Y. Then F_{X_n} = F_Y for all n \in \N, and hence \lim_nX_n = Y in distribution. If the common distribution F_Y is zero mean Gaussian with variance \sigma^2, then X_n - Y is zero mean Gaussian with variance 2\sigma^2. Therefore, for \epsilon < \sigma\sqrt{\pi} and all n \in \N It follows that P\set{\abs{X_n - Y} > \epsilon} \ge 1- \frac{\epsilon}{\sigma\sqrt{\pi}} for all n \in \N, and hence \lim X_n \neq Y in probability.

Lemma 4 (Convergence in probability implies in distribution). Consider a sequence X:\Omega\to\R^\N of random variables and a random variable X_\infty: \Omega\to\R defined on a probability space (\Omega, \sF, P), such that \lim_nX_n = X_\infty in probability, then \lim_nX_n = X_\infty in distribution.

Proof. Proof. We will show that all continuity points x of F_{X_\infty}, we have \lim_{n\to\infty}F_{X_n}(x) = F_{X_\infty}(x). Fix \epsilon > 0. Since x is a continuity point of non-decreasing function F_{X_\infty}, choose \delta > 0 such that F_{X_\infty}(x+\delta) - F_{X_\infty}(x-\delta) < \epsilon. Therefore, it suffices to show that For the chosen \delta >0, we consider the event A_n(\delta) \triangleq \set{\omega \in \Omega: \abs{X_n(\omega)-X_\infty(\omega)} > \delta} = \set{X_n \notin [X_\infty- \delta, X_\infty+\delta]}\in \sF, and define events A_{X_n}(x) \triangleq \set{X_n \le x} and A_{X_\infty}(x) \triangleq \set{X_\infty \le x}. Then, we can write

2 &A_X_n(x)A_X_(x+) A_X_(x+),&&A_X_n(x)A_X_^c(x+) A_n(),
&A_X_(x-)A_X_n(x) A_X_n(x), &&A_X_(x-)A_X_n^c(x) A_n().

From the above set relations, law of total probability, and union bound, we have From the convergence in probability, we have \lim_nP(A_n(\delta)) = 0, and the result follows. ◻

Theorem 5 (Central Limit Theorem). Consider an random sequence X:\Omega\to\R^\N defined on a probability space (\Omega, \sF, P), with \E X_n = \mu and \Var(X_n) = \sigma^2 for all n\in\N. We define the n-sum as S_n\triangleq \sum_{i=1}^nX_i and consider a standard normal random variable Y:\Omega\to\R with density function f_Y(y)= \frac{1}{\sqrt{2\pi}}e^{-\frac{y^2}{2}} for all y \in \R. Then,

Proof. Proof. The classical proof is using the characteristic functions. Let Z_i \triangleq \frac{X_i-\mu}{\sigma} for all i \in \N, then the shifted and scaled n-sum is given by \frac{S_n-n\mu}{\sigma\sqrt{n}} = \frac{1}{\sqrt{n}}\sum_{i=1}^nZ_i. We use the third equivalence in Proposition [Prop:ConvDistEquivalence] to show that the characteristic function of converges to the characteristic function of the standard normal. We define the characteristic functions

3 &_n(u) (ju), & &_Z_i(u) (juZ_i), & &_Y(u) (juY).

We can compute the characteristic function of the standard normal as Since the random sequence Z: \Omega\to \R^\N is a zero mean sequence, it follows that \Phi_{Z_1}^{(1)}(0) = j\E Z_1 = 0 and \Phi_{Z_1}^{(2)}(0) = j^2\E Z_1^2 = - 1. Using the Taylor expansion of the characteristic function \Phi_{Z_1}, we have For any u \in \R, taking limit n \in \N, we get the result. ◻

Strong law of large numbers

Definition 6. For a random sequence X:\Omega\to\R^\N defined on a probability space (\Omega,\sF,P) with bounded mean \E\abs{X_n} < \infty for all n \in \N, we define the n-sum as S_n \triangleq \sum_{i=1}^nX_i and the empirical n-mean \frac{S_n}{n} for each n \in \N. For each n \in \N, we define event

Theorem 7 (L^4 strong law of large numbers). Let X:\Omega\to\R^\N be a sequence of independent random variables defined on probability space (\Omega,\sF,P) with bounded mean \E X_n for each n \in \N and uniformly bounded fourth central moment \sup_{n \in \N}\E(X_n-\E X_n)^4 \le B < \infty. Then, the empirical n-mean converges to \lim_n\frac{\E S_n}{n} almost surely.

Proof. Proof. Recall that \E(S_n-\E S_n)^4 = \E(\sum_{i=1}^n(X_i-\E X_i))^4 = \sum_{i=1}^n\E(X_i-\E X_i)^4 + 3\sum_{i=1}^n\sum_{j \neq i}\E(X_i-\E X_i)^2\E(X_j-\E X_j)^2. Recall that when the fourth moment is bounded, then so is second moment. Hence, \sup_{i\in \N}\E(X_i-\E X_i)^2 \le C for some C \in \R_+. Therefore, from the Markov’s inequality, we have It follows that the \sum_{n \in \N}P(E_n) < \infty, and hence by Borel Canteli Lemma, we have Since, the choice of \epsilon was arbitrary, the result follows. ◻

Theorem 8 (L^2 strong law of large numbers). Let X:\Omega \to \R^\N be a sequence of pair-wise uncorrelated random variables defined on a probability space (\Omega, \sF, P) with bounded mean \E X_n for all n \in \N and uniformly bounded variance \sup_{n\in\N}\Var(X_n) \le B < \infty. Then, the empirical n-mean converges to \lim_n\frac{\E S_n}{n} almost surely.

Proof. Proof. For each n \in \N, we define events F_n \triangleq E_{n^2}, and From the Markov’s inequality and union bound, we have Therefore, \sum_{n \in \N}P(F_n) < \infty and \sum_{n \in \N}P(G_n) < \infty, and hence by Borel Canteli Lemma, we have The result follows from the fact that for any k \in \N, there exists n \in \N such that k \in \set{n^2, \dots, (n+1)^2-1} and hence ◻

Theorem 9 (L^1 strong law of large numbers). Let X:\Omega\to\R^\N be a random sequence defined on a probability space (\Omega,\sF,P) such that \sup_{n\in\N}\E\abs{X_n} \le B < \infty. Then, the empirical n-mean converges to \lim_n\frac{\E S_n}{n} almost surely.